112 research outputs found
Majority Rule: better patching via Self-Consistency
Large Language models (LLMs) can be induced to solve non-trivial problems
with "few-shot" prompts including illustrative problem-solution examples. Now
if the few-shots also include "chain of thought" (CoT) explanations, which are
of the form problem-explanation-solution, LLMs will generate a "explained"
solution, and perform even better. Recently an exciting, substantially better
technique, self-consistency [1] (S-C) has emerged, based on the intuition that
there are many plausible explanations for the right solution; when the LLM is
sampled repeatedly to generate a pool of explanation-solution pairs, for a
given problem, the most frequently occurring solutions in the pool (ignoring
the explanations) tend to be even more likely to be correct! Unfortunately, the
use of this highly-performant S-C (or even CoT) approach in software
engineering settings is hampered by the lack of explanations; most software
datasets lack explanations. In this paper, we describe an application of the
S-C approach to program repair, using the commit log on the fix as the
explanation, only in the illustrative few-shots. We achieve state-of-the art
results, beating previous approaches to prompting-based program repair, on the
MODIT dataset; we also find evidence suggesting that the correct commit
messages are helping the LLM learn to produce better patches
Improving Few-Shot Prompts with Relevant Static Analysis Products
Large Language Models (LLM) are a new class of computation engines,
"programmed" via prompt engineering. We are still learning how to best
"program" these LLMs to help developers. We start with the intuition that
developers tend to consciously and unconsciously have a collection of semantics
facts in mind when working on coding tasks. Mostly these are shallow, simple
facts arising from a quick read. For a function, examples of facts might
include parameter and local variable names, return expressions, simple pre- and
post-conditions, and basic control and data flow, etc.
One might assume that the powerful multi-layer architecture of
transformer-style LLMs makes them inherently capable of doing this simple level
of "code analysis" and extracting such information, implicitly, while
processing code: but are they, really? If they aren't, could explicitly adding
this information help? Our goal here is to investigate this question, using the
code summarization task and evaluate whether automatically augmenting an LLM's
prompt with semantic facts explicitly, actually helps.
Prior work shows that LLM performance on code summarization benefits from
few-shot samples drawn either from the same-project or from examples found via
information retrieval methods (such as BM25). While summarization performance
has steadily increased since the early days, there is still room for
improvement: LLM performance on code summarization still lags its performance
on natural-language tasks like translation and text summarization.
We find that adding semantic facts actually does help! This approach improves
performance in several different settings suggested by prior work, including
for two different Large Language Models. In most cases, improvement nears or
exceeds 2 BLEU; for the PHP language in the challenging CodeSearchNet dataset,
this augmentation actually yields performance surpassing 30 BLEU
A Survey of Machine Learning for Big Code and Naturalness
Research at the intersection of machine learning, programming languages, and
software engineering has recently taken important steps in proposing learnable
probabilistic models of source code that exploit code's abundance of patterns.
In this article, we survey this work. We contrast programming languages against
natural languages and discuss how these similarities and differences drive the
design of probabilistic models. We present a taxonomy based on the underlying
design principles of each model and use it to navigate the literature. Then, we
review how researchers have adapted these models to application areas and
discuss cross-cutting and application-specific challenges and opportunities.Comment: Website accompanying this survey paper can be found at
https://ml4code.github.i
- …